1. Field
Certain aspects of the present disclosure generally relate to neural system engineering and, more particularly, to a method and apparatus for strategic synaptic failure and learning in spiking neural networks.
2. Background
Learning in spiking neural networks typically involves adjusting a strength (weight) of a synaptic connection based on a learning rule such as the spike-timing-dependent plasticity (STDP). Over time, the synaptic connection strength may therefore change considerably. Moreover, the synaptic strength may be depressed to zero. At first glance, in a software simulation or hardware implementation of a spiking neural network, this should motivate disconnecting the zero-strength synapse to save computation involving an input that will have no effect on the receiving neuron.
However, there are several problems with this approach. First, the capability for structural changes (disconnecting or reusing a synapse for another connection) may be limited. Second, by disconnecting a synapse, the possibility for future learning (increase in strength of the connection) is rendered impossible. For example, even if the synapse strength is zero, the STDP may invoke a future increase in that synapse's strength due to post-synaptic firing. Third, depending on the learning rule and learning context, a portion of weights may converge toward zero but not necessarily reach zero or may reach zero but not stay there or cluster near zero. This presents a further difficulty on how to decide when to disconnect a synaptic connection in order to save computations and yet not to impact learning or network behavior.
The above problems have not yet been solved because either synapses are typically limited in number (and thus have limited capability/potential for learning) or models take longer time to simulate and/or run. For example, in biology, a synaptic failure is typically viewed as simply probabilistic (e.g., equal to a predetermined percentage, such as 40%). The synaptic failure can also occur deterministically if, for example, synaptic vesicles are not replenished with neurotransmitter. The present disclosure provides solutions to the aforementioned problems.